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ABSTRACT ' . . 

A teaching aid appropriate for a beginning course oa 
axperivental design is presented. The aid is a nulmerical example 
»hich iJllustrates some of the theoretical interrelations among three 
sompetling design analysis strategies for Estimating treatment effects 
in rajdom assignment designs. The analysis sl^ategies considered are 
analysis of variance (ANOVA), of the potettes.t, >KPV& of an index of 
eesponEe, and analysis of covariance (teoVA) . "ThVjEirst relationship 
illustrated is that all three strategies estimate thf same effects, 
rhus, choice of strategy may rest on precision and other factors no^t 
considered here, such as^ assumptions and robustness to violation _ of , 
assumptions. The second relationship illustrated was precision; it 
yas considered by comparison of mean square errors* Tables present a 
summary of the analysis strategies showing effects and error terms 
symbolically, as well as the numerical exaaELle itself 
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Introduction i , 

The purpose of this paper is to present 9 ntimerical example which < illus- 
trates some of the theoretical interrelationships among iseveral competing 
analysis strategies. The analysis S'trategies considered are ANOVA of the 
posttest', ANOVA of an indeit of response, and ANCOVA. The context for compar- 
ing the analysis strategies is any balanced design having random ass ig^ent 
of experimental units to levels of a fixed independent, variable (T), a singly ^ 
random dependent variable (Y), and a single random covariab la (X). , The num- 
erical example and the comparison 1 of results from competing analysis strate- 
gies may facilitate students' undirstanding of more general relationships. 

' One of the' topics typically conis^idered in a beginning course on experi- 
mental design is methods for improving precision, w4iere precision ^is defined 

1 ■ - *» . • ■"■ 

as the standard error of a simple contrast. It Is helpful to introduce these ^ 
methods for random assignment experiments so thatVhe more difficult problems 
of measuring change are not distracting. Thus, the studei.-: starts with' the 
belief that effects estimated by ANOVA o^ the post test are of-interest and 
that they may be used to define the null hypotheses that motivated the study.' 
A treatment effect for ANOVA of the posttest is defined 



* Invited paper for Educational Statisticians Symposium, "Tips for Teaching. 
Basic Stati'stical Concepts", AERA 4978. 
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where ^ ' denotes the population mean of the T level of the Independent 

T. ' # ' . 

variable on the posttest. The null hypothesis can then, be st^^.d ' 

• t" 

Ho: E 4 " ° ' 
, T»l T. 

where t denotes the number of levels of T. , - " 

... • ^ /V 

ANOVA of the posjttest serves as a baseline against which other strate- 
gies may be judged. In judging the utility of a new strategy the student 
yflrst^fcust be convinced that the effects estimated by the new strategy are 

le ones of interest., i.e», the same would be yielded by ANOVA of the post 
te^y. Once the. validity of the effects has been established, the student ^ 
►should consider precision. There are, of course, other criteria for ^elect- 
ing among design/analysis strategies, e.g., assumptions and robustness to 
violation of assumptions, but they are beyond the scope of this paper. 

Alternative Strategidfi . 

!. ■ ■ '"^ . ' ' 

ANOVA of indek of response is a design/analysis strategy that might be 

considered in p lac e^of ANOVA of the posttest* An index of response Is de- 

fined ' . ■ 

. . ^TR " ^TR ■ ^TR V 

where R denotes replication and K is any constant. The most popular form 
of index of response is when K equals one, i.e.*, a gain score« The design 
for ANOVA of index, of response is identical to that for ANOVA of posttest 
except that information is- also available on a covariable X. . Once the index 
has been created it is used as^the dependent variable in ANOVA. The effects 
estimated by ANOVA of Z are 



and are easily, obtained by_3ubstitutioh. Since given random ^ssl^nment all . 

.■ • 

' .. # ♦ • . . 

^ are zero» the student quickly sees that independent of choice of K>. the,. 

■ ^- •■ ■■' • ■ • . '\ ' ■ 

strategy estimates the desired effects. Relevant to pre<!lslony .the expected 

value pf the mean square error for ANOVA of Z is 

1 ■■ ■ . 

2 2. 2 2 2 

where. a == K-Py x'^XY correlation of X and Y and p^ is the slppe of Y 

on X^pooled within levels of T (Por^r and Chlbucos, 1973). ,^lle this ex- . 
pression makes it easy to see that setting K = p„ yields 'minimum error var- 

X • A 

iance^ it is also true that ANOVA (^f Z will be morfe precise than ANOVA of Y 
whenever / v . 

- • • - . ^ , / . 

Yet a third design/analysis strategy for improving precision is the 
analystis 6f covarf^nce. An ANCOVA effect is defined 

where prime denotes adjusted effect. As was true for. index of response, the 
second term of an ANCOVA* effect disappears Under random assignment and ANCOVA 
is s^iien to estimate the effects of interest. The expected value of ^he mean 
square error for ANCOVA is 

2 2 . r, . ' 1 
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wherje d.f. denotes degrees of freedom. 

Teaching ANOVA of index of response prior to ANCOVA has some advantages. 
First, ANOVA of index of response is an intuitive procedure which is compu-i 
tafcionally straightforward. "ANCOVA is computationally mysterioua but can 



. A 



be thought of as essentially an index of response in vjhlch ^^.X 
mated from the sample rather than known a priori . The smaller the sample 
the poorer the estimate of Pyx* ®^ larger [1 4- ^ g ,2 ] ^^^^ 
error vasjlance. Conversely, ^as sample size goes to, infinity, the sample 
estimate of x ^^"^^^^8^8 on the parameter, and the ^rror variance reduces 
to Its minimum. Thus, a second advantage of teaching ANOVA of index of re- 
sponse first is that, consistent with intuition, the limiting ^orm of the 
error variance for ANCOVA is identical to the minimum error variance for 

index of response. A summary of the effects and variances of Jthe three 

o • V • ■■ ' ' S • • 

strategies considered is provided- in Table 1. I ' ' 

■ ■ ; • ■ t ■ . 

A Numerical Example ' 

Table 2 "coptains data that can be used to illustrate the int;er relation-, 
ships presented in Table 1. In the example, there are two; treatments and two 
levels of a second independent variable, say sex. (S) , crossed with ^treatments. 
The design is balanced with five individuals p6r cell. The data in Table 2-. 
have. several properties which- facilitate instruction. First, the observations 
on the posttest (Y) and the covariable (X) are all whole numbers of modest 
size. Second, the correlation between X and Y is exactly .8 within each cell. 
Third, the within cell variances of X and Y are equal so that the slope of 
Y on X is 'equal to the correlation between Y and X. Finally, the treatment 
level means op the covariable are exactly equal. While this i& unlikely giveii 
random assignment, it is consistent with the long run expectation and will 
facilitate illustration of the -Table 1 relat;ionships<. ► ' -1. / 



* ANCOVA is not, liowever, computationally equivalent to ANOVA of Z Y - py.Y 

' where ft„ „ is the sample esti* ate of .slope. . 
'■^Y'X 



Ignoring the covdrtriable information it is possible to Analyze the post- 
test data using a two-way fixed eff'S^cts ANOVA. The results of such an analysis 
are reported in Table 4. • The F *te«t for a sex by treatment interaction is 
zero prolriding no evidence for rejecting ehe null hypothesis. The F test. for: 
a treatment effect is 2 which does not exceed the tabled critical value bf . , 
4.49 for ^ .OS. While the F test for a sex toain effect was significant, 
it, was ^rp'bably never in doubt. From the point of>^view of precision, the 



treatment effects 



A 



+ .5 



(Table S') were not^ judged to be" different jdrom. zero iisliig an error variance 

^ ■ _ 

Ttie ob&ervations on -contained in Table 2 might have been used to form 
an index of response. . Since gain scores have been so popular and because. ^ . 
they have;, also been so heavily criticized (Cronbach and Purby, 1970) gain 
scores is a pedagogically good starting points The results of ah ANOVA of 
gain scores are reported in Table d. Consistent with the conclusion from ; 
table 1 that index of response tests the same hypothesis as AKIOVA of the post- 
test, the' mean square* treatmejnt is equal to five in both Tables 4 arid 5. Simi- 
iarly, using the means from Table 3 the treatment effects for gain scores are 

. .■ ■> " ■ ' • 



T- , 



= + .5. 



It must be remembered, however, that this identity of effects at the sample 

^ ■• ' ■ • . [ ■■ . 

level i^ a function of exactly equal treatment means. on X and 'should not - 
be expected in practice. 



Using gWln scores the Interaction F test remains zero but the treatment 
effect. F test is 5 which is statistically significant at^ ^ .OS* Hie sig- 
nificant F results from the errdr variance having been reduced from 2*5 to 

• . - 

1.0. Ihe< main point for. the student Is that these results could have been * 



predicted from the relat 



Lonshlps Summarized In Table 1, i.e.» 

2 , 2)T2v 



2 



2 



- . . - 2.5 (1 - .8^) + (1 - .8)^2.5 ■ ^ 

; . ■ ' T . ' • • •"■ 

■ - 1.0 ■ 

Table 6 provides. Information on the- rela-tlonshlp between K/Bv y 
** / * ' ^ ' . 

'the mean' square error. Consistent wl^h the relationships given in Tttblel^ 

error variance is seen to be a minimum when K/^^^^j^ " ^ • Further, the example 

Illustrates that f Native to improvement: over ANOVA of the posttest the break- 

even points are K/b„ ^ equal 0 and 2, . 

As an aside, it is useful to point out to the student that the precision 

of gain scores for testing effects was quite gbod despite popular criticism 

about their low fell£iblllty. Explaining thje paradox is beyt>nd the scope of , 

" . .-. ■ ^ * ■ ■ ' • ■ "X i 

this paper but worth including in a- course on experimental design (Ijorter, | 
1973). The beginning s'tudent might also be interested to know that on occasion 
a repeated measures „ design has been suggested as an alternative to the low 
reliability gain scores. Whil6 the algebra is tedious » the numerical example 
from Table 2 provides an easy illustration that the two procedures are Iden-^ 
tlcal« Using ]|>retest as level one and posttest as level, two of a Repeated 
tqea^ures dimension crossed with T» S, and I; the ANOVA results are presented 
in Table(7. .The me^n squares In Table 7 are exactly half-^the comparable me^n 
squares for gain scores given In Tables 5 so* that the factor cancels when forming 



the F tests. . '> 

Analysis o£ (^variance la the final analysis strategy to be Illustrated 
using the data from Table 2 and the results are presented In Table. 8. Again, 
the mean square for treatments and treatments by sex Interaction are as , , 
before, 1-e., five and zero respectively. Similarly, ANCOVA adjuisted treat-*. ^ 
ment effects estimated using means from Table 3 are,-^ .5 as they were for each 
of the other design/analysis strategies* The mean square error for ANCOVA is 
^een to be *96 which could have been predicted from Table 1, I.e., 



^Y.X ''.^^Y ' PxyV[^ d.f.-2 ]. 

• 1 • 

= 2.5 (1 - .8^) [1 + 14] 

• » 

. ' = .96/" ■ 



Again, these numerical results allow the student to verify ^edlctlons ba^ed 
on relationships summarized In Table^ 1. For exad^le, the ANCOVA mean square 
error Is not as small as the minimum mean ;k'quare error for ANOVA of Index* 
of response. Further, not counting the loss of one degree of fri^eclom, ^n 
the example ANCOVA Is comparable ta gain scores In precis ion. At .this points 



the student may wish to refer to Cox (1957) wher<^an Index o^jirec 



8 Ion Is 



provided that reflects both mean square error and degrees of "freedom. . . 

Id summary, the purpose of this paper has been to present a teaching aid 
tl\at might be used In a beginning course on toperlmental design. The Intent 
o'f the aid is to provide concrete illustrations of some .of the interrelation- 
sliips among three .alternative design/analysis strategliea -for estimating treat- * 
ment effects In randopi assignment designs. The fiirst relationship to be' 
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Illustrated is that ^l - three strategies estimate the same effects. Ilius, 
choice of strategy inay\- rest on precision and other factors not considerad 



it 



here. The second relationehlp to be illustrated was precision find was ponsl- 
dered by cotnparison of mMn. square errors/' 
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ANOVA of Z 
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ANCOVA - 



Table 1 

Sunnary'of Analyati Strategies 



Effect . 



\ 



Error 



V ■ 
« • . 

/ 



\ 



2/1 2\ J. .2 2 * 



s . 




r 



V, 



10 



4 



Table 2 
Ex«inpl« . Data 




T 



id T«.; Tr^satnents Oaa and TWo 

I*- 2 ^ 7 

^ si i|fid S^: Two Sexes 

.J^;-vl2Qt Individuals . 

J- ^: j^ntecedent Information (covariabla) 
.Y: Pojpttes^^'C^ Outcome Variable' 
Y-X: Cain Scor^' 



Table 4 

Analysis of Variance of the Fosttest Data 



Sources 



d.f . 



SS 



MS 



T 


1 


5 


5 


2 


S 


1 


80 


80 


32 


TS 


1 


0 


0 


0 


I:TS 


16 


AO, 


2,5 





F-Value Required For Stattsticai Significance at a - .05 vith 
1 and .16 degifees of freedom la 4.49 • ^ 



V. 



' V 



' ErIc : 




.\ 



— Tab le 5 



Analysis of Variance of the 


Gain 


Scores 


* 


Sources 


d.f. 


SS 


MS 


, * 
F 


/. 


;■ 1 


5 


5 


5 


y = ' ■ 


1 


u 


□ 






1 


0 


0 


/o 


I:TS 


.16 


lie ° 

i 
1 


1 




1'. 

* Required 


F =.4.49 (d.f* 1, 


16- 


a =^ .05) 


■7 



V 



\ f 

\ i 
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Table 6 

The Effect of Choice of K on the Pre- 
cision of ANOVA of Iridex of Response 

\ ■ ■ ■ • 

\ 2 



-.2 


-.25 


3.4 


.0 


.00 


2.5 


.6 


.75 


1.0 


.8 


' 1.00 


.9 


I. a 


1.'25 


1.0 




2.00 


2.5 


1.8 


2.25 


3.4 





Table 7 



Repeated M^sures Analysis of Variance 
Using FretesJ: and Post test as Measures 



Sources 


d.f. 


SS 


MS 




T 


K - 

1 








S 


1 








I:TS 


. 15 








M 


■ 1 








, TM 


1 


2.5 


.2.5 


5 


SM 


. 1 


0 


b 


0 


TSM 




0 


0 


0 


Ifi:TS" 


16 ' 


8 


.5 





F,> 4.49 required for significance at a =• -Q5, with, l-and t&si.jE. 
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Table 8 ' 

Analysis of Covariance of th^ Post test Data Using X as the Covariate 



Sources 


d.f . 










d.f.* 




T 


1 


0 


5 


0 




1 


5 


S'" 


1 


80 


80 


80 


1. 07 ' 


1 


1.07 


TS 


1 


0 


0 


0 


0 


1 




I :TS 


16 


AO 


40 


32 


14.4 


15 


.96 



5.21 
1.11 

of . 



F must equal or exceed 4.54 to be statistically slgnifleanjt mt^a • .05* for 
i and 15 d.f. / - , 



-n 
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